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ABSTRACT 

' Osing ef f ectivemess, efficiency, self-sustenance, and 
communicability as criteria, a conceptual model, called SCRAPE, was 
developed at the University of Texas to systematically describe 
educational behaviors. The key elements of the system are: (1) 
diagnosis and prescription, (2) instructional events, (3) achievement 
evaluation, and (U) consequent activities. Within each of these 
elements, behaviors can be classified as either input processing, 
recording, monitering, or output. Having outlined a course in this 
fashion, an instructor could then ^use the assistance of an 
instructional engineer to determine which teaching processes sho)ild 
be human and which should substitute technology; the engineer could 
then take the responsibility to design the nonhuman elements of the 
educational system. This report offers a paradigm for developing 
educational systems, and it includes detailed examples of the 
variables that could effect each element of the system. (EMH) 
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THE SCRAPE MODEL: A CONCEPTUAL APPROACH TO EDUCATIONAL EVALUATION 



Paul G. Liberty, Jr. 

This paper presents background considerations on the design of 
educational systems and then presents a conceptual model (SCRAPE) to guide 
educational program-planning and evaluation. It specifies the kinds of 
considerations and the kinds of data collection procedures needed to evaluate 
educational projects^ Some specific variables are presented to illustrate 
how to «ake operational and how to use the SCRAPE model. 

Although the SCRAPE model focuses upon computer-based education 
(C-BE) applications, it has planning and evaluation applicability for any 
educational or social action project. The value of the model is that it 
specifies the "elements of possible concern." It allows the investigator 
to identify potential variables (which can be very diverse in educational-social 
programs) and then consciously to decide which variables will be included in the 
evaluation design. Following these d^i^ons, the investigator can specify 
statistical designs and plan the analyses. 

An Educational System 

Instructional technology has primarily been defined in terms of student 
behavior. In the February 1968 issue of the National Society for Programmed 
Instruction Journal . Don Tosti commented on the absence of individualized in- 
structional systems that were behaviorally based and were both practical and 
self-sustaining. In defining the objectives of any operational system, Tosti 
formulated the PRIME (PRescriptioW; Interaction, Motivation, Evaluation) model, 
which was at the time being employed at the Westinghouse Learning Corporation 
in Albuquerque. The remaining portion of this section is quoted (with comments 
added concerning Project C-BE applications) from Tosti, even though the PRIME 
model was not conceived with the computer in mind.* 



*Tosti, Donald T. Prime—A general model for instructional systems. NSPI 
Journal, 1968, Vol. VII (2), 11-15. 
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System Objectives 



Any practical educational system must be: 

1. effective 

2. efficient and economical 

3. self-sustaining . 

4. communicable. 

Effectiveness > Effectiveness is defined in terms of learner 
achievement. Both the relative achievement of the individuals and 
the number of individuals who achieve the educational objective 
must be considered. 

... If a sizable portion of the learner population does not 
achieve the stated objectives, the program should be modified or 
abandoned. Experimental systems may be allowed to operate below 
th^_expected criterion for a short time, but such systems eventually 
must come up to or exceed proposed [italics added] standards. 

In addition, the results of the experimental program must inevitably be com- 
pared against the results of other competing programA^^ (The above points out 
that both criterion-referenced testing, involving expected standards of per- 
formance, and norm-referenced testing, involving comparisons between individ- 
uals and programs , are required to ascertain program effectiveness.) 

Efficiency and Economics . In the design of the system, considera- 
tions of economy and efficiency can be determined only after some 
educational objectives or function of the system are specified. 
Once this has been accomplished, refinements can be viewed as 
changing the ratio of objectives met or functions accomplished 
to a learner-time base, a teacher-time base, or a dollar base. 
An example of the resulting dependent variables might be reading- 
gain per learner hour or learner assignments per instructor hour. 

Self-Sustaining Operations . It is important that the system be 
constructed in a form that will allow experimental personnel to 
be withdrawn ftom the actual operation without disrupting the 
program. Only by such a procedure can it be determined if the 
operation is independent of the skill and art of those involved 
In its organization. This step is the first requisite for re- 
producibility [and transferability]. 

Coimnunicability . Implementation is a necessary component of 
any applied system. The key problem in implementation is the 
ease with which the system can be communicated. This requires 
two strategies. First, the procedures used in the system must 
be refined to make them as clear and easy to learn as possible 
and, if possible, be related to a general model. Second, since 
the system necessitates a change in teacher activities, teacher 
training units must be developed. 
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The PRIME Model 



Research into classroom management and instructional materials 
has led Westinghouse Learning Corporation (WLC) to formulate a gen- 
eralized four-part model for the classroom. [the PRIME model is 
shown in Figure 1 below]. 



(1) 
Placement 
[Diagnosis,] and 
Prescription 



(2) 

Instructional 
Events 



(3) 

Achievement 
Evaluation 



(4) 

Consequent 
Activities 



Figure 1 



Components 1 and 3 are derived in part from the application 
of the technology of repertoire assessment developed in the testing 
and guidance fields. 

i Component 2 is concerned with the technology of presentation, 

an area to which people in programed instruction have contributed 
a wealth of information. Unfortunately, those who are knowledgeable 
in the technology of presentation have been somewhat unconcerned 
about the work of researchers involved with educational systems 
development, and there is a definite tendency for the systems 
people to belittle the technology of the programers. 

Often, the time allotted to program writing is considerably. underestimated,. 

* Component 4 can be considered an application of the technology 

of motivation, which has been primarily identified with the con- 
tingency management system. . . . 

Within the C-BE program, there would appear to be a number of different strat- 
egies that instructor/developers might employ to motivate students (e.g., no 
final exams, no term papers, "work as you want," etc.). Any evaluation should 
assess student motivations for signing up for a C-BE project course and rea- 
sons for completing or not completing it. 

This breakdown of components may not exhaust all possible im- 
portant factors. .. [However, it is possible] to describe many of the 
gross characteristics of these [fdarj components and describe the 
methods that have been employed. 

Placement atid Prescription ^ . . ♦The primary objective of the prescriptive 
process is to direct the learner through those materials necessary for 
him to achieve the desired objectives. 

The prescriptive method usually involves two phases — initial 
assessment and differential assignment* 

* The initial assessment attempts to list existing characteristics 

of the learner which will be useful in determining future assignments. 
The most common characteristic examined is present achievement level • 
In some systems, such as those used by the Job Corps, this is the only 
measurement used to initially place learners. [Some C-BE project 
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directors may elect to use measures of] interests, aptitudes, or 
learning styles to aid in assignraentt . . 

Prescriptive methods have some resemblance to techniques 
used in branching programs, but the simplest form of "pure" 
prescription is found in multi-level programs.... Each section 
of such a course is designed in a multi-level amplification for- 
mat. The highest and smallest level of amplification consists 
of a set of self-prescriptive pre-test items covering each 
section ':s content. The learner is not expected to be able to 
pass at. this level unless he is using the course for review 
purposes; or is, for some other reason familiar with the 
section's content. If he can pass this prescriptive pre-test 
and the corresponding post-test, he is directed to the pre-test 
of the next Section. 

The second or intermediate level of amplification consists 
of a narrative summary with a corresponding intermediate pre- 
scriptive tefet. In most cases, the- reader will find that he 
can pass a number of sections at this level. If not, he will 
be directed ta the third level which is written on a still more 
amplified level.... 

This form of prescription^ allows the teacher to assign 
certain instructional units, exercises, or supplementary 
activities to the learner based on his pre-test achievement 
results, and the teacher's knowledge of the materials, and 
the learner's past performance. Some branching and linear 
programs have used short tests to skip learners ahead. An 
extension of this logic is found in prescriptive methods which 
measure whole knowledge areas. Thus, a learner who is weak in 
a specific arithmetic skill, such as division of fractions, may 
not have to learn addition of fractions (Unless he tests poorly 
in this area too). The resulting assignment should include 
only those skills which he requires. 

Prescriptive tests may take the form of pre-tests or post- 
tests, or may be imbedded in the curriculum. Usually the pre- 
test type is used to ascertain the existing skill-4reas of the 
learner or any other behavior which may facilitate the acquisition 
of new skills. These initial measurements indicate an appropriate 
starting point and also aid in future assignments by identifying 
areas of strength or weakness. 

General techniques employed in the construction of achieve- 
ment tests are applicable. The difference between diagnostic 
tests and prescriptive tests lies in the latter 's division 
into behavioral clusters which consider the terminal and inter- 
mediate objectives. The subject matter is analyzed and divided 
into many sub-areas or steps according to the skills needed to 
emit appropriate responses from the entry level to the terminal 
objective. 

The outcome of a prescriptive pre-test may lead directly 
to a curriculum decision or may ^indicate the need for more ^ 
precise testing. Flowcharts may be employed as devices to * 
aid. the instructor to make accurate prescriptions as shown \ 
in Figure 2. 
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Prescriptive Test Results (Test Form POl) 
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Figure 2 

Figure 2 indicates that some of the results tell the 
teacher to assign a particular instructional unit. Other 
results indicate further testing. 

The primary difference between prescriptive tests and 
other evaluation devices is that these tests are used to 
' make curriculum decisions. They are not used for learner 
control or for grading purposes. The control function is 
handled by the progress check test (see achievement evalua- 
tion). No grade should be given on these tests, and the 
learner need not be directly aware of the results. 

Probably only a few C-BE projects will have courses permitting "multiple 
access points." However, as test items are written to assess the specific 
course objectives, the possibility of assembling test items to yield pre- 
scriptive as well as evaluative information certainly becomes available. j 

Instructional Events . Several of the operating programs employ 
combinations of individualized and group instruction. It is 
important to note that the objectives of the PRIME model may 
be accomplished by either method. 

The decision as to which subjects will be presented in 
individualized form and which by group instruction [e^g., 
rear-screen projection] is dependent in part on whether good 
individualized material is available, the cost of preparing 
or purchasing such materials compared with present efficiency 
and the occasional need for the teacher to assume the lecture 

role little [if anything] has been done to determine what 

learning factors would lead to the decision to have courses 
ot parts of [C-BE] courses [taught in lecture]... or a seminar 

discussion form the union of this technology [of presentation] 

with practical considerations of the classroom is a pressing 
need. 

There are several types of individualized instructional 
materials commercially available. The Job Corps has been able 
to assemble a basic remedial, education curriculum composed only 
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of individualized materials . This curriculum was drawn from 
.extant material and only slightly revised for the Job Corps* 
Most of these materials are linear programed texts, reading 
laboratory packages, and workbooks • 

Achievement Evaluation * Most individualized instructional 
systems provide for sqme form of achievement feedback control.,.. 
The presence of frequent achievement assessment tends to control 
the 1 earner *s behavior by the knowledge that he soon will be 
tested on what he is studying. 

Many recent programs contain progress check tests. The 
progress check is a short test covering a small segment of 
material. It is not only used for control purposes but has 
d:efinite motivational properties. Often in individualized 
instruction, it is difficult for the learner to see his pro- 
gress. The progress check, which allows the learner to demon- 
strate his newly acquired skills, aids in alleviating this 
problem. This demonstration of mastery is often the only 
formal, positive consequence employed in some individualized 
instructional systems* 

Progress checks have a number of possible uses. Several 
of the prescriptive oriented systems employ progress checks to 
make further differential assignments. This usually results 
in reams of evaluation records that can be reasonably handled 
only by a computer. In the contingency management system, 
progress checks ares employed as indicators of task completion, 
signaling to the learner that he now may engage in some pre- 
ferred activity. i 

In programed njaterial, which has not been revised to in- 
clude progress checks, additional work must be done to adapt 
them to the PRIME System. For example, several linear programs 
designed to teach ^English cover the subject matter well. But 
the units are generall^^too long, and the learner has no oppor- 
tunity to demonstrate his ability until he has covered as many 
as 1,Q00 frames. In experimental work, the introduction of 
progress checks has i-ncreased the efficiency of this program. 

Consequent Activities , The establishment of sound, lasting 
motivation is at once the most important and the most difficult 
step in building and applying an effective system of instruction. 
The initial interest intrinsic in most educational programs often 
wanes after the first few hours of instruction. Full attainment 
of the learning objectives designed into any given self- instructional 
program is very difficult unless proper motivation is provided. 

Motivation is of utmost .importance in all areas of education. 
No matter how well-conceived, organized, and behaviorally sound a 
presentation may be, students will learn nothing from it unless 
they attfend and respond. Yet, until recently, there existed no 
formal method of reliably producing learner motivation. 

Motivation is often loosely termed the energfzer of behavior; 
it is said that individuals are motivated to eat, to sleep, to make 
money, to be famous, etc. However, such a broad definition does 
not aid in determining which methods wij.1 maintain activity in 
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classroom learning tasks. Therefore, it is necessary to limit 
this definition to a more specifiable condition. When an indi- 
vidual initiates and completes a given task, it might be said 
that he has "task motivation." If the task is the construction 
of a retaining wall, and the learner completes it, he has task 
motivation. If the task is the completion of a page of graded 
reading material, and the learner completes it satisfactorily, 
he has task motivation. Task motivation, then, is a descriptive, 
summarizing term for the complex of behaviors displ ayed by a 
learner when he is attending to and responding approp riately to 
his^ given task assignment . [Italics added.] 

Task motivation is particularly important in individualized 
instructional systems where there is no teacher or peer pressure 
to maintain performance. Generally speaking, programed instructional 
materials are published with no specific directions or recommendations 
to control motivation. The learning results of these materials are 
generally good when learners complete them, but the rate of completion 
is often extremely slow. 

To keep the student in the learning environment or to keep him 
responding at a normal rate, his learning activity must lead to some 
preferred consequence. Laboratory studies of reinforcement typically 
use the preferred consequence of eating or drinking to motivate animal 
behavior. Equivalent kinds of reinforcement can be used with children. 
However, it is awkward to use this kind of payoff for learning ac- 
tivities in the classroom. Humans cannot be starved, nor can candy 
be placed in their mouths for correct responses. This creates an 
impossible position: rewarding consequences are necessary but im- 
practical. 

Those who will object to the simple treatment of behavior and 
reinforcement presented here may argue that human motivation is a 
highly complex affair, involving much more than the simple matter 
of the consequences that behaviors produce. This may be a valid 
premise, but it surely is not a sound reason for ignoring the 
simple facts of life. Acceptance of the fact that human motiva- 
tion is affected by individual emotions, ideas, hates, and loves, 
should not rule out also accepting the importance of the conse- 
quences of behavior. 

The formal administrative technique employed to provide positive 
consequences for all learning activities has been termed "contingency 
management." Contingency contracts may be oral or written. 

The following procedures have been developed by Westinghouse 
Learning at the Capital Job Corps Project. The names of available 
courses are typed on plain 3" by 5" cards. These are called task 
cards and are filed in boxes in the front of the room. The task 
cards are yellow. Blank, white cards represent "breaks." The 
classroom manager explains in simple terms that the system will 
help trainees learn if they organize the cards so that a task 
they like occurs after a task they do not care for. The follow- 
ing guidelines are placed on the contract by the classroom manager. 
Each contract must contain at least three tasks, and the contract 
is to be completed before the trainee may leave class for the day. 
There is no restriction on either the number of breaks or the max- 
imum number of tasks. Each day the trainee prepares a new contract,,,< 
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Upon completion of a predetermined task phase of his . 
contract, the Cprpsman may take a break. He reports to 
the classroom manager that he is ready for a break, and 
then selects from a group of cards numbered from 5 to 20 
in increments of five. ,The card he selects determines his 

* break time. Heathen goes to the RE [reinforcing event] area. 

There he can play cards, smoke, read magazines, talk to his 
friends, sleep, sit, or get coffee. When his break is over, 

* he returns to his study room to complete the contract. 

Within the C-BE Projects, directors may Sesire to devise and try out certain 
activities that will promote and perpetuate "task ^notivation." Since computer- 
based education presents students with new task requirements, it may be nec- 
essary for projects to determine what degree or "student control," ^^eriodi'C 
feedback to students, and student choice is important to learning outcomes. 
Higher-^than-conventional dropout/incomplete rates in some C-BE courses and 
disruptive personality effects within some C-BE students indicate a need to 
relate students' motivational/personality characteristics against subsequent 
course perfomance in C-BE projects. 

I [End of citation from TostiJ 

^ A Behavior Systems Approach 

From the description of the PRIME model, it can be seen that a 
behavioral system rather than an instructional system has been posited. 
Behavior systems are networks of interrelated behavioral events which act 
upon certain kinds of input variables to transform them into particular 
output variables. The emphasis thus is on behavior, i.e., functions, acts, 
and results. The physical entities that produce these behaviors may be 
human or machine. 

Five Categories 

The function of the behaviors in the system may be divided into 
five major categories: input behaviors, processing behaviors, recording 
(storage or memory) behaviors, monitoring behaviors, and output behaviors. 
Each one of these classes of behaviors may constitute a major subsystem of 
the behavior system. Evaluation takes place within each of these categories 
and may be labeled as input, process, and product types of evaluation. Con- 
text (needs assessment) evaluation is omitted here. 
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It is a prime concern of behavior systems engineers to define 
objectives of the behavior system as a whole in terms of the products that 
the system i6 to generate from the given input. Such a definition will 
follow the sentence-function, "Given input I, the system will produce out- 
put 0," where appropriate substitutions for the variables I and 0 will 
generate the' objective sentence. Temporal, quality, and other criteria 
may be added, but the general form of the objective sentence will not 
change. 

Behavior EnglneerlnR 

Within C-BE projects, once the project director or other staff 
decides whatjthe "desired output will be when a given input is made, the 
component designers are responsible for determining what components are 
necessary to produce each of the specified behaviors. The components 
(there are at least five in innovative education projects) may J)e human 
or Tion-huihan, and they may be readily available or may have to be developed. 
Some of the behaviors can be produced by a single person or device. Others 
may have to be produced by a combination of several persons and/or devices;' 
in the case of C-BE, this usually involves developing a human component [the 
student through a special device" (the computer),, or supplementing an available 
human's skills (the teacher's) with appropriate devices (the computer and 
simulation assignments)]. At a simpler level, the optimal solution may be 
to add an electric calculator to a system instead of teaching arithmetic 
operations to the human. Both short- and long-term objectives of the be- 
havior system will determine the optimal solution in most cases. 

Then, after the director has specified the objectives and the 
components designers have stated what the particular component should be, an 
instructiona\ engineer will have the task of developing necessary non- 
human aspects while the behavioral engineer is responsible for the de- 
veldpmant. of the human aspects. Both instructional .(non-human) and be- 
havioral (human) engineers have to develop behavioral specifications 
that will meet the component design and, the overall behavioral system 
design. 
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It Is not only necessary, therefore, for C-BE projects to focus 
on the preparation of educational materials (subject matter), but also to 
eake into^GQunt the total systems objectives which include the characteris- 
tics of the learner, teacher, the curriculum (materials), and the socio- 

educational setting in which the desired behaviors are to occur. 

\ 

A Conceptual Paradigm for Evaluation 

A potentially useful schema for conceptualization of the variables 
to be considered in an evaluation design is shown in Figure 3 below. ^ 

SCRAPE Model /' 

' The model emphasizes student (learning) outcomes (Lg) and the 
various, factors that are presumed to affect student performance in C-BE 
courses . ; 



\ 



Student Entering 
Characteristics 



Lg = f (S,C,R,A,P,E, 

(OUTPUT (INPUT CHARACTER- 
BEHAVIORS) ISTICS)^ I 



Curricular (FOCAL) 
Characteristics 



CBEj^ , CBE^ , CBE^ , CBE^ , CBE^ , 

(INSTRUCTIONAL PROCESS 
CHARACTERISTICS) 

Figure 3 



Educational Environment 
(BACKGROUND) Characteristics 



^Et'Ets'^d^Vpc'V 

(CONTEXT CHARACTERISTICS) 



Outcome (Dependent Variable) Behaviors 



L = learning behavior (outcomes) of individual student to be described, 
S 

explained or predicted, such as: 



(1) 
(2) 
(3) 
(4) 
(5) 
(6) 
(7) 
(8) 
(9) 



level of score on content po^ttest; 
whether or not criterion was reached; 
comparison with conventional (and C-BE) courses; 



student attitu^de change; 
student perso;nality change; 
enhanced mental abilities; 
change of major; 

intent to take another C-BE course; 

intent to take another course in same department or related area. 
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Student Entering Characteristics 
S - Social class characteristics of student 

C - Cultural (Anglo, Black, Brown, etc.) characteristics of student 

R - Readiness of student for computer-based education 

A = Abilities ^nd aptitudes of student 

P = Personality characteristics of student 

E = Educational success and academic background of student 

Curriculum Characteristics 
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CBEj^ = Type of content emphasis in course 
CBE^ = Type of computer i^sed in C-BE course 
CBEj = Presentation mode in C-BE course 
CBE^ = Entry skills to course 
CBEj = Interim, or unit, scores* 

Educational Environment Characteristics 



E^ = Teacher (instructor) characteristics. 
Erpc ~ Teaching style of instructor. 

Ej^ = Departmental attitudes toward C-BE and prior C-BE experiences of 

faculty and students in department. 

E^ = College attitudes toward and experiences with C-BE. 
\j 

E-,p = Previously existing course and prior educational experiences of students 

It v» 

prior to C-BE course (control groups) . 
= Educational milieu of C-BE course: where, when, how, whar,; time of 
day of course; time required, if any, to spend on computer. 

Although admittedly simplistic, the paradigm does provide directors^, cur- 
riculum developers, and evaluators with a way to approach program planning 



* A particular course might have several di8cer|iible or distinct components or 
units. The intent in CBE^ and CBE^ is to focus on prior ;Learning, both prior 

to the course and during the course prior to end-of-course evaluation of 
learning, for purposes of pupil-learning evaluation and curriculum effective- 
ness evaluation. 
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and program evaluation conceptually. It attempts to serve the purpose of 

identifying crucial variables that might affect performance. The paradigm 

has been found useful ini "getting all the cards on the table" so that the 

entire staff, often busy with only part of the picture, can discuss the full 

range of possible activities in the program. The paradigm can be expanded, 

of course, through the addition of variables or the specification of finer 

variables. For example, personality characteristics may be enumerated 

P to P . Also, the paradigm permits systematic selection and/or develop- 
1 n 

ment of tests and other types of instrumentation. 

/ 

Additionally, the paradigm lands itself to conceptualizations of 
what variables are to be treated in a research-evaluation-statistical design. 
For example, the typical research design might exclude all variables except 
CBE and E^^ (the prior curriculum), thereby providing comparisons of student 
effects between the new C-BE curriculum and the previously existing curriculum. 
A more enlightened design might include several levels (high, medium, low) of 
A (student academic ability) in a comparative study of both types of curriculum. 
Another design variation might include levels cf E (prior academic success, 
i.e., GPA) of students, P (sex of students), and R (readiness of student, 
e.g., motivation, attitudes, attitude toward machines, fear of computer, etc.) 
for C-BE instruction. 

Finally, at the level of evaluation across C-BE projects, the paradigm 
affords administrators, committees, and evaluators a number of "conceptual 
handles," or conceptual discriminators, that may help to contrast and compare 
the separate prajects. 

SCRAPE and Other Outcome Variables 

Although the primary output, or effects, variable in C-BE projects 
is student learning, the 'model can just as easily treat o^rher variables as 
outcome (dependent) variables. Personality change in'^tiyients might be one 
such variable, where the concern is to explore whether ^3E alters certain 
personality traits, such as task-orientation and af f ili^tional-orientation 
of students. Also, CBE^, secondary outcomes (e.g., scoAre on a later cur- 
riculum unit) might be investigated by employing CBE^ levels of performance 
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(90% and above, 80-89%, and 70-79%) on a criterion test covering CBE^ 
content. (Does performance on the CBE^ criterion test relate to dif- 
ferential learning outcomes on CBE^?) And, finally, for the curricu- 
lum developer, curriculum unit CBE^, or its parts, would be investigated 
for evidence that students are meeting criterion standards on this unit. 

When the curriculum developer finds that certain types of students 
fail a unit one or more times, he may desire to examine the program inter- 
nally for existence of certain types of difficulties that students indicate 
on tests or call to his attention; he may need to. rewrite some portion of 
the curriculum. He or the evaluator may observe or interview students to 
determine the nature of problems encountered by the students. Or, the 
curriculum developer may have found that specified criterion standards 
were met and go on to further developmental activity, while the evaluator 
may feel compelled to determine the characteristics of the 20% of the 
students who did not meet criterion—in other words, what distinguished 
these students from those who reached criterion standards sooner. The 
evaluation information obtained may contribute important information 
for the curriculum designer, and the information may be extremely im- 
portant in identifying and subsequently shaping the. crucial factors 
that are necessary for the introduction and implementation, here and 
elsewhere, of C-BE programs. Thus, the evaluation function is a con- 
tinual study for expected and unexpected outcomes and a search to 
identify reasons, at least initially, through ex post facto analyses. 

It may be apparent that both product and process evaluation 
exist within the SCRAPE model. Process evaluation, simply stated, is 
concerned with documentation and feedback processes that permit a project 
to know what things (expected and unexpected) are happening throughout 
the year. The process evaluative function is concerned to a great extent 
with identifying critical information easily so as to facilitate program 
correction and alteration. Product evaluation (pre- and post-testing) 
during the development years of a project is important but not sufficient 
to insure that a program is operating effectively. Thus, process evaluation 
is especially important in action-research situations in which a program 
is underway and any alterations, or oth^ decisions, must be made before 
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the end of the school year* 

Some Possible Variables 

\ 

In order to begin estimating the magnitude of data to be collected 
within a C-BE project, an attempt is made below to specify variables that 
seem important enough for inclusion primarily for purposes of Product 
Evaluation. 



Student Learning Outcome Variables 



Immediate and Related Academic Variables 



\ 



(a 
(b 
(c 
(d 

(e 
(f 
(8 
(h 
(i 
(j 

(1 
(m 
(n 
(o 
(P 

(q 

(r 



Grade la course (1 digit) 

Proportion of course ^jjntent learned digits) 
T^hether or not specified criter|.on was reached (1 digit) 
Number of semester hours taken during present semester, exclusive 
of C-BE course (s) (2 digits) 

Number of semester hours in any other C-BE course (s) (2 digits) 
Semester GPA, all non-C-BE courses (3 digits) 
Semester GPA, all C-BE courses (3 digits) 
Semester GPA, all courses (3 digits) 
Incomplete in C-BE courses (1 digit) 
Incomplete in non-C-BE course (s) (1 digit) 
.Dropped C-BE course, passing (1 digit) 
Dropped C-BE course, failing (1 digit) 
lJumber of non-C-BE course (s) dropped; passing (1 digit) 
Number of non-C-BE course (s) dropped, failing (1 digit) 
Major is different from major stated before course started (1 digit) 
Majojr changed to be in C-BE course area at end of semester when 
C-BE course taken (1 digit) ^ 
Pre-test in course (comprehensive) (3 digits) 
Post-test in course (comprehensive) . (3 digits) 



-Concurrent Variables 

(a) Took advanced exam in another course in area of C-BE course. 
(1 digit) 



IS 



-15- 



(b) Took and passed advanced exam in 'another course in area of 
C-BE course, (1 digit) 

(c) Evidence of development in a C-BE course of 

(1) a level in a C-BE course taught skill that exceeds level 
in regular course. (2 digits) 

(2) a level of a skill that is achieved in a C-BE course but 
' usually is not obtained in non-C-BE course. (2 digits) 

(d) Major is different from stated major at start of C-BE course. 
\ (1 digit) 

(e) \ Major is different at start of a C-BE course but is declared 
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to be in the C-BE course area at end of the semester when C-BE 

I * 
course was taken. (1 digit) 

(f) ) Course-Instructor Survey ratings of: (50 digits) 

(1) Course . \ ' . 

(2) Instructor \' 

(3) Computer assisted instruction methodology \^ 

Long-term Variables 
(Variables still to be specif ied,> 

Student Characterlstilcs Variables 

Social class (S) (1 digit) 

Cultural group (C) (1 digit) 

Readiness of student for C-BE (R) : a questionnaire (2 digits) 

Abilities and aptitude measures (A) 

(a) High school GPA (3 digits) 

(b) High.school rank^- (2 dkgits) 

(c) High school courses taken in C-BE related course area (2 digits) 

(d) High school GPA in. CrBEhrelated course area (3 digits) 

(e) SAT-verbal score (3 digits) 

(f) SAT-mathematics score (3 digits) 

(g) SAT-total score (4 digits) 

\ 
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, Personality Characteristics (P) 

(a) Sex (1 digit) 

(b) Age (2 digits) 

(c) Orientation Inventory (Bass) 

(1) Task orientation (2' digits) 

(2) Affiliation (person) orientation (2 digits) 

(3) ^ Self-orientation (2 digits) 

(d) Orientation to College 

(1) Vocational (1 digit) ^ 

(2) Intellectual (1 digit) 

(3) Academic-social (extra-cutricular) (1 digit) 

(4) Self-identity seeking (1 digit) 

(e) Machine vs, man scale (2 digits) 

(f) Reasons for taking C-BE course (26 digits) 

(g) Expected grade in course - C-BE (1 digit) 

(h) Expected grade in course - standard (1 digit) 

(i) Number of hours spent on computer, total (2 digits) 
(j) Number of hours spent on computer in: 

(1) First six weeks' (2 digits) 

(2) Second six weeks (2 digits) 
, (3) Third six weeks (2 digAs) 

(k) "Volunteer" vs, "non-volunteer" (1 digit) 

(1) Degree of expected liking for C-BE course (1 digit) 

> 

Educational Success and Background (t) 

(a) Academic classification (1 digit) 

(b) College in' which' registered (2 digits) 

(c) Major (2 digits) 

(d) College CPA (3 digits) 

(e) CPA in C-BE course area (3 digits) 

(f) CPA in C-BE-related coutse area (3 digits) 

(g) Major changes, number (l digit) 

(h) Prior. C-BE course taken (1 digit) 

(i) Previous exposure to CAI, C-BE (1 digit) 

(j) Diagnostic or prescriptive test level/ at which student begins 
instruction (3 digits) 
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Currlculum Characteristics (C-BE) 

Type of C-BE program emphasis (simulation, computation, etc.) (2 digits) 
Acadeinic level of C-BE course (1 digit) 
Subject Area of C-BE course (2 digits) 

Secondary component emphasis (e.g., math component to support 
physics) (4 digits) 

Scores on interim^or unit, tests (20 digits) 
Environment; Educational Setting Characteristics 
Teacher characteristics 

(a) Teacher or no teacher in course (i digit) 

(b) Graduate student or professor — rank (2 digits) 

(c) Sex of instructor (1 digit*) 

Teaching style of instructor 

(a) Type or level of interaction with students required in course. 
(1 digit) 

(b) Degree of interaction with students, from students' report - 
time (2 digits) 

(c) Degree of interaction with studfM»t-, from instructor log - 
time (2 digits) 

(d) Student rating of level (impo7;tance) of help provided by 
instructor (1 digi^t) 

Departmefttal attitudes toward C-BE ; questionnaire (2 digits) 

College administrators' attitudes toward C-BE : questionnaire (2 digits) 

Previous course (before C-BE course) information ^- — 
(a) Whether C-BE course is "replacement" course or new course 
(1 digit) 



/ 

'/ 

i 

(b) Data from previous course (s^ticms of it) : 

(1) Achievement of students (6 digits) 

(2) Ratings by students (6 digits) 

(3) Other information for comparisons (6 digits) 



Educational Milieu 

(a) Readiness to accept innovations in conununity (setting) ; 
pooled ratings of principals involved in innovating 

(3 digits) 

(b) Time of day C-BE course is taught (2 digits) 

(c) Required amount of time to be spent on' computer in hours 
(2 digits) 

(d) Classroom tests or computer-test (2 digits) 

\ 

(e) Other homework, term papers (2 digits) 

(f) Distance to computer for student (2 digits) i 

(g) Average waiting time required for students— experiential data 
(2 digits) 

(h) What did students do while waiting? (Study, t^lk about non- 
course things, talk about C-BE course matters.) (1 digit) 

(i) I^en were tests given? (1 digit) ^ 
(j) When were tests scored? (1 digit) 

(k) Was proctor present? (1 digit) 

Process Evaluation 

Process evaluation involves the collection of data for documentation 
and feedback. Its purpose is to trace developn^nts within the project for 
purposes of ascertaining whether or not objectives on student llearning and 
curriculum are being reached and to detect "hangups'* in the learning process, 
whether or not these derive from machine, students, lack of materials, in- 
accessibility,- etc. In other words, the question to.be answered is: "I«ia,t 
data are necessary for us to determine that our program Is on course, not 
only against previously stated, expected outcomes, but also from potential 
t)r actual unanticipated disruptions?" A feedback netx^ork should pick up these 
^problems; documentation is necessary so that subsequent efforts here and by 
others will be alert to pitfalls. 
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Process evaluation .will utilize a number of criterion variables - 
posited in the immediately preceeding section (Some Possible Variables) 
for product evaluation. The criteria will be applied during the course 
of the project instead of at the end of the project. 

Note again that objectives, both process and product, will exist, 
formally or informally, in those seven components that appear to be common 
to educational programs. These components are (a) instructional; (b) cur- 
riculum development; (c) materials/equipment; (d) staff development; (e) 
space-time structures; (f) community involvement — in general, the external 
relationships affecting and being affected by the project and (g) manage- 
ment: coordination of series of steps producing coordination of the internal 
components (a-e above) and the external world (f ) . These components were previousl 
specified in "Handout for Evaluation Seminar," Paper No. 2. 

Other Thoughts on Behavioral Model 

As previously indicated, the conceptual paradigm, SCRAPE, can be 
rewritten so that other variables besides student learning variables are 
treated as outcome (dependent) variables. Thus, a personality variable j 
may be investigated if it is hypothesized that C-BE exposure will change ' 
personality; for example, change aff illative orientation to task orienta- 
tion. 

A sheet that one project evaluator (Gavenda's project) believes to 
represent the minimum informiation to be obtained from each student completing 
a session at the terminal i^; attached as an appendix. 

A conceptual paradigm lends itself to formulation of multivariate 
statistical design. It (SCRAPE) allows for pre-post study if a comprehensive 
subject pre- test is .givdn, for experimental (C-BE) vs. control (non-C-BE) 
group studies, and for Comparisons between C-BE programs when non-content 
variables are of interest. 

Curriculum development checkout can be accomplished within the 
context of the model. Developers will probably want to relate studdnt 
performance on units to student results on earlier units. Ability levels 
of students and other variables might be employed to determine what var>- 
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ables affect learning and non-learning (criterion Veached or not reached) 
or levels of learning. Developers will probably wish to become vary familiar 
with item analysis procedures to help determine wha^^ items are causing prob- 
lems (and what kinds of problems) to students. Curriculum developers may 
also wish to create small experimental groups for tryout of materials. This 
procedure affords quick feedback and such in-depth feedback from students as may 
be needed to understand the students' problems. 

The model also can be applied with prescriptive-diagnostic-type 
programs wherein students are tests for the level at which instruction should 
begin. This concern can be subsumed under "Educational Background of Student." 




APPENDIX 
Developed by Ben Manny 

For each session: 
DATE (6 bits) day 
TIME (4 bits) nearest hour ' 
LESSON IDENTIFICATION (5 bits) - (32 lessons) 
Total correct time (9 bits) in minutes, 8 1/2 hours max. 
Total TM time (12 bits) in seconds, approx. 4096 max. 

4096 = 1 hour 
409*6 = 6.8 min. 

TYPE Of DAY (3 bits) 
Value of Session (3 bits) 



Main purpose of Session: Type in program 

Debut program 
Run debugged program 

Combination of above 



Classwork 

Play with machine 

Demonstrate machine 

to friends 
Own program work 



Type of Day Scale: 

1. 2. 

One of ray worst days; A bad day 

nothing has gone right 



3. 



An average day 



4. 

A good day 



5. 



One of my best days 



Value of Session: 
1. 

Ruined my day 
4. 

Learned a lot; 
time well spent 



2. 

Waste of time; 
gained nothing 

5. 



3. 

Average session; 
learned a little 



One of my best sessions; 

time could not have been spent better 
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